in_kubernetes_events: Efficiently stream kubernetes events via watch by ryanohnemus · Pull Request #8351 · fluent/fluent-bit

ryanohnemus · 2024-01-04T23:30:46Z

Change in_kubernetes_events plugin to watch kubernetes events after requesting the event list. Instead of polling for the full event list every 500ms (default), an initial full events list is requested and then is a watch is requested. The watch will create an efficient http chunked stream that will push events as they are added, modified, or deleted from the cluster. The interval_sec and interval_nsec plugin config options now act as a reconnect timer if the watch stream is ended, instead of timer to re-poll the k8s cluster.

Potentially Breaking: this will require the kubernetes role used by fluent-bit to have watch permission in addition to the current list and get permissions.

Fixes #8315

Leaving in draft as this is dependent on both #8316 & #8323, will rebase and move out of draft after those are reviewed/merged.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

[ X] Example configuration file for the change

[INPUT]
    name          kubernetes_events
    tag           k8s_events

Debug log output from testing the change

Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

Run local packaging test showing all targets (including any new ones) build.
Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Documentation required for this feature

Backporting

Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

plugins/in_kubernetes_events/kubernetes_events.c

ryanohnemus · 2024-01-10T18:07:37Z

@edsiper - just force pushed a new version of this. I still have this PR draft mode because it was branched off of 2 other PRs: #8316 & #8323. I was assuming it would be easier to review those each individually and then I'd rebase this one with a smaller set of changes to review, but if you'd prefer to just do them all in one change PR review/merge, I can move this pr to ready to review.

ryanohnemus · 2024-03-23T13:36:27Z

@edsiper @patrick-stephens this didn't get merged in with 3.0 but it is ready to go. Can this be added to the next milestone/release?

patrick-stephens · 2024-03-25T11:34:28Z

Did you carry out the review comments from @edsiper ?

ryanohnemus · 2024-03-25T11:59:15Z

@patrick-stephens yes, unless I missed something they should all be marked resolved. I rebased and force pushed after some dependencies had a merge conflict, i think messes with the comment visibility

edsiper · 2024-04-09T22:25:27Z

@pwhelan do you think you can take a look at this one ?

pwhelan

I have a lot of comments regarding the manner in which the events are continually processed.

plugins/in_kubernetes_events/kubernetes_events.c

ryanohnemus · 2024-05-01T14:39:36Z

Rebased and force pushed the following updates:

moved upstream and http client for stream to context to ensure we can clean this up on shutdown for no leaks
added 2 tests that are using monkey/server to mock the k8s upstream connection
chunked-streaming test is something that can be added later on, these tests were to show the no memory leak:
continuous wait on stream still exists, assuming this is ok since this the input is threaded

valgrind --leak-check=full ./bin/flb-rt-in_kubernetes_events 

[removed test output]

SUCCESS: All unit tests have passed.
==931== 
==931== HEAP SUMMARY:
==931==     in use at exit: 0 bytes in 0 blocks
==931==   total heap usage: 3,845 allocs, 3,845 frees, 2,002,206 bytes allocated
==931== 
==931== All heap blocks were freed -- no leaks are possible
==931== 
==931== For lists of detected and suppressed errors, rerun with: -s
==931== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

@pwhelan please take another look

(posting this before CI has fully run, so i will review any test failures if they show up)

ryanohnemus · 2024-05-01T15:56:07Z

appveyor and macos build errors both appear to be flakes.

ryanohnemus · 2024-05-31T16:25:40Z

@pwhelan @edsiper @lecaros - this fell out of the next milestone with the last few 3.0 releases but is still ready to go. Can this be added to the existing 3.0.7 milestone? Thank you

Potentially breaking change as it now requires the rbac used by fluent-bit to have 'watch'. Uses a k8s watch instead of http api polling to stream k8s events from the kube api server Signed-off-by: ryanohnemus <ryanohnemus@gmail.com>

Signed-off-by: ryanohnemus <ryanohnemus@gmail.com>

ryanohnemus · 2024-06-16T12:57:09Z

Rebased to fix merge conflict.

@edsiper @pwhelan @lecaros could this be tagged in the 3.1.0 milestone so it does not get missed? Thank you!

patrick-stephens · 2024-06-17T09:44:19Z

We probably should update docs as well, particularly with the RBAC change. Could you link a docs PR @ryanohnemus ?

Do we have any int tests for this btw?

ryanohnemus · 2024-06-19T11:39:00Z

@patrick-stephens added doc via fluent/fluent-bit-docs#1396

No int tests, but I added unit tests for the plugin in this PR.

github-actions bot added the docs-required label Jan 4, 2024

ryanohnemus temporarily deployed to pr January 4, 2024 23:31 — with GitHub Actions Inactive

ryanohnemus mentioned this pull request Jan 4, 2024

http_client: Add Ability to Process Http Chunked Stream #8316

Merged

7 tasks

ryanohnemus temporarily deployed to pr January 4, 2024 23:49 — with GitHub Actions Inactive

edsiper requested changes Jan 9, 2024

View reviewed changes

ryanohnemus force-pushed the feature/in_k8s_watch branch from 14b804c to f4093d7 Compare January 10, 2024 17:51

ryanohnemus temporarily deployed to pr January 10, 2024 17:52 — with GitHub Actions Inactive

ryanohnemus temporarily deployed to pr January 10, 2024 18:13 — with GitHub Actions Inactive

ryanohnemus requested a review from edsiper January 10, 2024 19:53

edsiper added this to the Fluent Bit v3.0.0 milestone Jan 10, 2024

ryanohnemus force-pushed the feature/in_k8s_watch branch from f4093d7 to 8809b42 Compare January 16, 2024 20:07

ryanohnemus temporarily deployed to pr January 16, 2024 20:08 — with GitHub Actions Inactive

ryanohnemus force-pushed the feature/in_k8s_watch branch 2 times, most recently from 8639849 to 589a7cb Compare January 16, 2024 20:16

ryanohnemus temporarily deployed to pr January 16, 2024 20:17 — with GitHub Actions Inactive

ryanohnemus temporarily deployed to pr January 16, 2024 20:37 — with GitHub Actions Inactive

ryanohnemus force-pushed the feature/in_k8s_watch branch from 589a7cb to 9e61635 Compare January 19, 2024 13:29

ryanohnemus temporarily deployed to pr January 19, 2024 13:29 — with GitHub Actions Inactive

ryanohnemus temporarily deployed to pr January 19, 2024 13:50 — with GitHub Actions Inactive

ryanohnemus temporarily deployed to pr March 20, 2024 12:54 — with GitHub Actions Inactive

ryanohnemus temporarily deployed to pr March 20, 2024 13:15 — with GitHub Actions Inactive

ryanohnemus force-pushed the feature/in_k8s_watch branch from aefe2d6 to c473bd9 Compare March 23, 2024 12:37

ryanohnemus temporarily deployed to pr March 23, 2024 12:38 — with GitHub Actions Inactive

ryanohnemus temporarily deployed to pr March 23, 2024 13:02 — with GitHub Actions Inactive

lecaros modified the milestones: Fluent Bit v3.0.0, Fluent Bit v3.0.2 Apr 5, 2024

edsiper modified the milestones: Fluent Bit v3.0.2, Fuent Bit v3.0.3 Apr 16, 2024

pwhelan reviewed Apr 29, 2024

View reviewed changes

plugins/in_kubernetes_events/kubernetes_events.c Show resolved Hide resolved

lecaros modified the milestones: Fluent Bit v3.0.3, Fluent Bit v3.0.4 Apr 29, 2024

edsiper mentioned this pull request Jun 5, 2024

in_kubernetes_events: refactor time check to use struct flb_time. #8887

Merged

7 tasks

ryanohnemus added 3 commits June 15, 2024 15:20

in_kubernetes_events: stream k8s events via watch

3719d5b

Potentially breaking change as it now requires the rbac used by fluent-bit to have 'watch'. Uses a k8s watch instead of http api polling to stream k8s events from the kube api server Signed-off-by: ryanohnemus <ryanohnemus@gmail.com>

in_kubernetes_events: add tests and move connection stream up to context

18efb7f

Signed-off-by: ryanohnemus <ryanohnemus@gmail.com>

in_kubernetes_events: check http_get_response_data return val

f526aa4

Signed-off-by: ryanohnemus <ryanohnemus@gmail.com>

ryanohnemus mentioned this pull request Jun 19, 2024

in_kubernetes_events: add k8s permission details fluent/fluent-bit-docs#1396

Merged

BrewTestBot mentioned this pull request Jul 9, 2024

fluent-bit 3.1.0 Homebrew/homebrew-core#176771

Closed

Conversation

ryanohnemus commented Jan 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryanohnemus commented Jan 10, 2024

Uh oh!

ryanohnemus commented Mar 23, 2024

Uh oh!

patrick-stephens commented Mar 25, 2024

Uh oh!

ryanohnemus commented Mar 25, 2024

Uh oh!

edsiper commented Apr 9, 2024

Uh oh!

pwhelan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ryanohnemus commented May 1, 2024

Uh oh!

ryanohnemus commented May 1, 2024

Uh oh!

ryanohnemus commented May 31, 2024

Uh oh!

ryanohnemus commented Jun 16, 2024

Uh oh!

patrick-stephens commented Jun 17, 2024

Uh oh!

ryanohnemus commented Jun 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ryanohnemus commented Jan 4, 2024 •

edited

Loading